TLDR: we are making progress on the Rust implementation of the GNU coreutils.
Well, it is an understatement to say
my previous blog post interested many people. Many articles, blog posts and some podcasts talked about it! As we pushed
coreutils 0.0.12 a few days ago and getting closer to the 10 000 stars on github, it is now time to give an update!
This has brought a lot of new contributors to this project. Instead of 30 to 60 patches per month, we jumped to 400 to 472 patches every month. Similarly, we saw an increase in the number of contributors (20 to 50 per month from 3 to 8). Two new maintainers (Michael Debertol & Terts Diepraam) stepped in and have been doing a much better job than myself as reviewers now! As a silly metric, according to github, we had
5 561 clones of the repository over the last 2 weeks!
The new contributors focused on:
- Performances. Now, some binaries are significantly faster than GNU (ex: head, cut, etc)
- Adding missing binaries or options (see below)
- Improve the testsuite: we grew the overall code coverage from 55% to 75% (in general, we consider that a 80% code coverage on a project is excellent).
- Refactoring the code to simplify the maintenance. Examples:
- Using the same code for permissions for chgrp and chown
- Managing error the same way in the various binaries - (Kudos to Jeffrey Finkelstein for the huge work)
- Improving the GNU compatibility (thanks to Jan Verbeek, Jan Scheer, kimono-koans and many others)
- Move to clap 3. Upgrade by Terts which unblocks us on various problems.
- ...
Closing the gap with GNU
As far as I know, we are only missing
stty (change and print terminal line settings) as a program.
Thanks to some heroes,
basenc,
pr,
chcon and
runcon have been implemented. For example, for the two last programs, Koutheir Attouchi wrote new crates to
manage SELinux properly. This crate has been used for some other utilities like cp, ls or id.
Leveraging the GNU testsuite to test this implementation
Because the GNU testsuite is excellent, we now have a proper CI using it to run the tests. It is pretty long on the Github action CI (almost two hours to run it) but it is an amazing improvement to the way we work. It was a joint work from a bunch of folks (James Robson, Roy Ivy III, etc). To achieve this, we also made it easier to run the GNU testsuite locally with the Rust implementation but also to ignore some tests or adjust some error messages (see
build-gnu.sh and
run-gnu-test.sh).
Following a suggestion of Brian G, a colleague at Mozilla (he did the same for some Firefox major change), we are now collecting the history of fail/pass/error
into a separate repository and generating a daily graph showing the evolution of regression.
At this date, we have, with GNU/Coreutils 9.0:
| Total |
611 tests |
| Pass |
214 |
| Skip |
84 |
| Fail |
298 |
| Error |
15 |
We are now automatically identifying new passing tests and regressions in the CI.
For example:
Warning: Congrats! The gnu test tests/chmod/c-option is now passing!
<br />Warning: Congrats! The gnu test tests/chmod/silent is now passing!
<br />Warning: Congrats! The gnu test tests/chmod/umask-x is now passing!
<br />Error: GNU test failed: tests/du/long-from-unreadable. tests/du/long-from-unreadable is passing on 'master'. Maybe you have to rebase?
[...]
<br />Warning: Changes from master: PASS +4 / FAIL +0 / ERROR -4 / SKIP +0
This is also
beneficial to GNU as, by implementing some options, Michael Debertol noticed some incorrect behaviors (
with sort and
cat) or an uninitialized variable (with
chmod).
Documentations
Every day, we are generating the user documentation and of the internal coreutils.
User documentation:
https://uutils.github.io/coreutils-docs/user/ Example:
ls or
cp
The internal documentation can be seen on:
https://uutils.github.io/coreutils-docs/dev/uucore/ For example, the backup style is documented here:
https://uutils.github.io/coreutils-docs/dev/uucore/backup_control/index.html
More?
Besides my work on Debian/Ubuntu, I have also noticed that more and more operating systems are starting to look at this:
In parallel,
https://github.com/uutils/findutils/, a rust dropped-in replacement for find, is getting more attention lately! Here, the graph showing the evolution of the program using the
BFS testsuite (much better than GNU's).
What is next?
- stty needs to be implemented
- Improve the GNU compatibility on key programs and reduce the gap
- Investigate how to reduce the size of the binaries
- Allow Debian and Ubuntu to switch by default without tricky manipulation
How to help?
I have been maintaining a l
ist of good first bugs for new comers in the repo!
Don't hesitate to contribute, it is much easier than it seems and a terrific way to learn Rust!
The following contributors got their Debian Developer accounts in the last two months:
I built my last desktop computer what seems like ages ago. In 2011, I was in a
very different place, both financially and as a person. At the time, I was
earning minimum wage at my school's caf to pay rent. Since the caf was owned
by the school cooperative, I had an employee discount on computer parts. This
gave me a chance to build my first computer from spare parts at a reasonable
price.
After 10 years of service